On Family Rigidity Theorems I 



Kefeng LIU and Xiaonan MA 

■ - - . Abstract. In this paper, we first prove a local family version of the Atiyah-Bott- 

Q>y ', Segal-Singer Lefschetz fixed point formula, then we extend the famous Witten's rigidity 

0> I Theorems to the family case. Several family vanishing theorems for elliptic genera are 

also proved. 
O 

w . Introduction. Let M, B be two compact smooth manifolds, and tt : M -^ B he 

OO i a submersion with compact fibre X. Assume that a compact Lie group G acts fiberwise 

on M , that is the action preserves each fiber of vr. Let P be a family of elliptic operators 
O ' along the fiber X, commuting with the action G. Then the family index of P is 

Q' 
■3 



a 



(0.1) Ind(P) = KerP - CokerP G KdB). 



Note that Ind(P) is a virtual G-representation. Let chg(Ind(P)) with g E G he the 
equivariant Chern character of Ind(P) evaluated at g. 

In this paper we will first prove a family fixed point formula which expresses chg(Ind(P)) 
^ ' in terms of the geometric data on the fixed points X^ of the fiber of vr. The by applying 

^ ■ this formula, we generalize the Witten rigidity theorems and several vanishing theorems 

Q . proved in [Liu4] for elliptic genera to the family case. 

O i A family elliptic operator P is called rigid on the equivariant Chern character level 

Q^ I with respect to this S^-action, if chc,(Ind(P)) G H*{B) is independent oi g E S^. When 

0\ ' the base P is a point, we recover the classical rigidity and vanishing theorems. When B 

^ • is a manifold, we get many nontrivial higher order rigidity and vanishing theorems by 

c^ . taking the coefficients of certain expansion of chg. For the history of the Witten rigidity 

theorems, we refer the reader to [BT], [K], [L], [H], [Liul] and [Liu2]. The family 
vanishing theorems generalize those vanishing theorems in [Liu4], which in turn give 
is^ ; us many higher order vanishing theorems in the family case. In a forthcoming paper, 

j_j ■ we will extend our results to general loop group representations and prove much more 

general family vanishing theorems which generalize the results in [Liu4]. We believe 
there should be some applications of our results to topology and geometry, which we 
hope to report on a later occasion. 

This paper is organized as follows. In Section 1, we prove the equivariant family 
index theorem. In Section 2, we prove the family rigidity theorem. In the last part of 
Section 2, motivated by the family rigidity theorem, we state a conjecture. In Section 3, 
we generalize the family rigidity theorem to the nonzero anomaly case. As corollaries, 
we derive several vanishing theorems. 



1 Equi variant family index theorem 

The purpose of this section is to prove an equivariant family index theorem. As pointed 
out by Atiyah and Singer, we can introduce equivariant families by proceeding as in 
[ASl, 2]. Here we will prove it directly by using the local index theory as developed by 
Bismut. 

This section is organized as follows: In Section 1.1, we state our main result. Theorem 
1.1. In Section 1.2, by using the local index theory, we prove Theorem 1.1. 

1.1 The index bundle 

Let M, B be two compact manifolds, and vr : M — > i? be a fibration with compact fibre 
X, and assume that dimX = 2k. Let TX denote the relative tangent bundle. Let W 
be a complex vector bundle on M and h^ be an Hermitian metric on W. 

Let h'^^ be a Riemannian metric on TX and V"^"^ be the corresponding Levi-Civita 
connection on TX along the fibre X. Then the Clifford bundle C{TX) is the bundle 
of Clifford algebras over M whose fibre at a; G M is the Clifford algebra C(T^X) of 
(TX,/i^^). 

We assume that the bundle TX is spin on M. Let A = A^ © A^ be the spinor 
bundle of TX. We denote by c(-) the Clifford action of C{TX) on A. 

Let V be the connection on A induced by V"^"^. Let V'^ be a Hermitian connection 
on {W, h^) with curvature R^ . Let V^®^ be the connection on Ai^W along the fibre 
X: 

(1.1) V^®^ = V®1 + 1®V^. 

For b e B, we denote by Eb, E±^b the set of C°^-sections oi A ^W, A± (g) W over 
the fiber X^. We regard the Eb as the fibre of a smooth Z2-graded infinite dimensional 
vector bundle over B. Smooth sections of E over B will be identified to smooth sections 
oi A(g)W over M. 

Let {cj} be an orthonormal basis of (TX, h^^), let {e*} be its dual basis. 

Definition 1.1 Define the twisted Dirac operator to be 

(1.2) ^^ = E.c(e.)V^®^. 

Then D^ is a family Dirac operator which acts fiberwise on the fibers of vr. For b & B, 
D^ denote the restriction of D^ to the fibre Eb. D^ interchanges E^ and E^. Let D^ 
be the restrictions of D^ to E±. By Atiyah and Singer [AS2], the difference bundle over 
B 

(1.3) Ind(D) = KerD+,6 - KerD_,fe. 

is well-defined in the i^-group K{B). 

Now, let G be a compact Lie group which acts fiberwise on M. We will consider 
that G acts as identity on B. Without loss of generality we can assume that G acts on 
{TX, K^-^) isometrically We also assume that the action of G lifts to A and W , and 
that the G-action commutes with V^. 



In this case, we know that Ind{D^) G Kg{B). Now we start to give a proof of a local 
family fixed point formula which extends [AS2, Proposition 2.2] 

Proposition 1.1 There exist Vj E G with j = l,---,r, a finite number of sections 
(sj.+i, ■ ■ •, Sj +i) with ij+i ~ij = dim V^- ofC°^{B, E^) such that we can find a basis {cj^i} 
of Vj, under which the map D+^b : C°°(S, E+,5) © ©J^iV^- -^ C^{B, E^^b) given by 

(1-4) D^f,{s + T^j^iXj^iCj^i) = D^s + J:j^i\j^iSi^+i 

is G-equivariant and surjective. The vector spaces KeTD_^_,^ form a G-vector bundle 

KerD_,_ on B, and the element [KerD_|_] — ©j^iV^- G Kg{B) depends only on D^ and 
not on the choice of {Vj} and the sections {si}. 

Proof: Given bo G B, we can find a > and a ball U{bo) C B around 60, such that 



X,2 



for any b G U{bo), a is not an eigenvalue of D 

Let eI '"^ = E^^j^^^E^'j^^ be the direct sum of the eigenspaces of D^ ' associated to the 
eigenvalues A G [0,a[. By [ BeGeV, Proposition 9.10], E^'^'"'^ forms a finite dimensional 
sub-bundle E^'^l C E over U{bo). Clearly, E[°'"[ is a G-vector bundle on U{bo). By [S, 
Proposition 2.2 ], we have an isomorphism of vector bundles on B 

(1.5) E[0'"[ = (Sy^QRomGiV, E[°''^[) © V, 

where G denotes the space of all irreducible representations of G. We can also find 
ti^k £ C°^(f/(6o), HomG(V^, -EI'" )) such that for b G U{bo), the elements ti^i form a basis 
of HomG(V^, E_ )b. Let {ci^i} be a basis of Vi. Then we can choose the sections ti^k^i^i G 
C°^{B, E ) to be our Sj's. This proves the first part of the proposition locally. 

The global version now follows easily by extending the above local sections of C°^(f/(6o), 
E-) together with a use of the partition of unity argument. This is essentially the same 
as the proof of [AS2, Proposition 2.2]. ■ 

By [S, Proposition 2.2], we have 

(1.6) Ind(D^) = ®y^QRouiG{V, Ind(Z^^)) © V 

and HomG(V',Ind(D^)) G K{B). We denote by (Ind(L'^))^ G K{B) the G-invariant 
part of Ind(D"^). 

By composing the action of G and the Chern character of HomG(V", Ind(D"^)), we get 
the equivariant Chern character chg(Ind(D"^)) G H*{B). 

Definition 1.2 We say that the operator D-^ is rigid on the equivariant Chern character 
level, if ch.g{hid{D-^)) is constant on g E G. More generally, we say D^ is rigid on the 
equivariant K- Theory level, ifhid^D'^) = (lnd{D^))'^ . 

In the rest of this paper, when we say D^ is rigid, we always mean D^ is rigid on the 
equivariant Chern character level. 



Now let us calculate the equivariant Chern character ch.g{Ind{D^)) in terms of the 
fixed point data of g. 

Let T^ M be a G-equivariant sub-bundle of TM such that 

(1.7) TM = T"M®TX. 

Let P^^ denote the projection from TM to TX. HUE TB, let U^ denote the lift of 
U in T^M, so that vr.f/^ = U. 

Let h^^ be a Riemannian metric on B, and assume that W has the Riemannian 
metric h'^^ = h'^^ © '7i*h'^^ . Note that our final results will be independent of g^^. 
Let V"^*^, V"^'^ denote the corresponding Levi-Civita connections on M and B. Put 
V'^'''" = p^^V"^^ which is a connection on TX. As shown in [Bl, Theorem L9], V"^"^ 
is independent of the choice of h'^^ . Now the connection V"'""^ is well defined on TX 
and on M. Let R^^ be the corresponding curvature. We denote by V and V'^®'^ the 
corresponding connections on A and A^W induced by V"^"^ and V*^. 

Take g E G and set 

(L8) M^ = {x e M,gx = x}. 

Then tt : M^ ^ B is a fibration with compact fibre X^. By [BeGeV, Proposition 6.14], 
TX^ is naturally oriented in M^. 

Let A^ denote the normal bundle of M^, then A^ = TX/TX^. We denote the 
differential of g by dg which gives a bundle isometry dg : N ^ N. Since g lies 
in a compact abelian Lie group, we know that there is an orthogonal decomposition 
N = N{tt) 00o<e<,r^(^)5 where dg\N(^j,) = -id, and for each ^, < ^ < tt, A^(^) is a 
complex vector bundle on which dg acts by multiplication by e*^, and dimA^(7r) is even. 
So A^(7r) also is naturally oriented. 

As the Levi-Civita connection V"^^ preserves the decomposition TM = TM^ ©o<9<7r 
A^(^), the connection V"^"^ also preserves the decomposition TX = TX^ Q)o<e<TT N{9) on 
M^. Let V^'''"'', V^, V^*-^^ be the corresponding induced connections on TX^, N and 
N(9), and et i?"^^", i?^, R^^^^ be the corresponding curvatures. Here we consider A^(^) 
as a real vector bundle. Then we have the decompositions: 

(1.9) /2'^^ = i?^^'' © R^, i?^ = ®eR'"''^. 



Definition 1.3 For < 6 < n, we write 



ch3(W^,V^) = Tr 









i dTXH 



An 



R 



:i.lO) ' ' ' Vsinh(^/2^^« 

Ae(Ar(^),V^(')) ^ 



i5dimiv(e)deti/2h _^exp(^i?^W)j 
Let ch.g{W), A(TX^), A0{N{9)) denote the corresponding cohomology classes on M^ . 



If we denote by {xj, —Xj} (j = 1, ■ ■ ■ , /) the Chern roots of N{9), TX^ such that Uxj 
define the orientation of N{6) and TX^, then 



"-' i I , 1 / •^ 7 ' 



A{TX3) = nj^/sinh( 



2 ^2 



We denote by tt^, : H*{M3) -^ H*{B) the intergration along the fibre X^. 
Theorem 1.1 We have the following identity in H*{B): 

(1.12) chg{lnd{D'')) = TT, {Uo^e<.MN{e))A{TXS)chg{W)^ . 

1.2 A heat kernel proof of Theorem 1.1 

As Atiyah and Singer indicated in the end of [AS2], we can proceed as in [AS1,2] to 
introduce an equivariant family, and then to find a formula for the equivariant Chern 
character of the index bundle. Here, we will use a different approach by combining the 
local relative index theory and the equivariant technique to give a direct proof of the 
local version of Theorem 1.1. 

We denote by °V = V^^ © tt* V"^^ the connection on TM. Let S = V^^ - °V. By 
[Bl, Theorem 1.9 ], {S{.)., .)^tm is a tensor independent of h^^. For U G T^M, we 
define a horizontal 1-form /c on M by 

(1.13) k{U) = Y,{S{U)e„e^). 

i 

Definition 1.4 Let V^ denote the connection on E such that if U E TB and s is a 
smooth section of E over B, then 

(1.14) V§s = V^Ts. 
If U, V are smooth vector fields on 5, we write 

(1.15) T(?7^, V^) = -P^^[f/^, V^] 

which is a tensor. 

Let /i, ■ ■ ■ , /m be a basis of TB, and /^, ■ ■ ■ , /™ be the dual basis. Define 

(1.16) c(T) = is.,,rA(T(/f,/,^)). 

Definition 1.5 Fort > 0, let At be the Bismut superconnection constructed in /Bl, %3], 

(1.17) At = VtD'' + (V^ + h) - ^^c{T). 



It is clear that At is also G-invariant. 

Let dvx denote the Riemannian volume element on the fiber X. Let $ be the scaling 
homomorphism from A{T*B) into itself : u -^ (2'n-i)^^'^^^^^^'^uJ. 

Theorem 1.2 For any t > 0, the form ^TYs[gexp{~A^)] is closed and that its coho- 
mology class is independent oft and represents chg(Ind(D"^)) in cohomology. 

Proof: Just proceed as in [Bl, §2(d)]. ■ 

Theorem 1.3 We have the following identity 
limi_o'^Tr,[^exp(-A2)] = 

Proof: If A is a smooth section of T*X(g)A(T*_B)(8)End(A(g)W^), we use the notation 

2k 

(Vj^^ + A{e,)f = 5^(Vf ^ + A{e,)r - V^r^..^, - A(E?^,Vj^e,). 

i — 1 e^ * 

Let V^ be the connection on A(T*B)^A^W on the fibre X. 

(L19) V; = V^«^ + ^ {S{.)e„ ff) c{e,)f- + 1 (^(.)/f , f^) f^f 

Let K^ denote the scalar curvature of the fiber (X, h^^). By the Lichnerowicz formula 
[Bl, Theorem 3.5], we get 

A't = -t{VtJ' + Ik^ + |c(e,)c(e,)i?^(e„ e,) 

(1.20) - 4 2 

+v^c(e.)/"i?^(e„ f^) + -r/i?^(/f , f^). 

Let Pu{x,x',b)(b G B,x,x' G Xb) be the smooth kernel associated to exp(— A^) with 
respect to dvx{x'). Then 

(1.21) $Tr,[(?exp(-A?)] = / ^TTs[gPtig-'x,x,b)]dvxix). 

Jx 

By using standard estimates on the heat kernel, for b & B, we can reduce the problem 
of calculating the limit of (1.21) when t — ;► to an open neighbourhood Us of X^ in Xh- 
Using normal geodesic coordinates to X^ in Xb, we will identify Us to an e-neighbourhood 
of X^ in Nx9/x- We know that, if (x, z) G Nxs/x with x G X^, then 

(1.22) g-\x,z) = {x,g~'z). 



Let dvxs{x), dvxxg/x X with x G X^ be the corresponding volume forms on TX^ and 
g/x induced by h^-^ . Let A;(x, z)(x G X^, 2; G Nxa/Xi \z\ < e) be defined by 



1^1.23) dvx = k{x,z)dvx9{x)dvNxg/xi^)- 



Then it is clear that 

k{x,0) = 1. 
By the discussion following (1.21), (1.23), we get 



:i.24) 



limt^o^Ti s[gexp{-A^)] = limt_o /^^ ^Tr^ gPt{g ^x,x 



dvx (x) 



y&Nx9/x 

dvx9{x)dvNx,^x(X)- 



gPAg-\x,Y),ix,Y)] k{x,Y) 



By taking Xq G X^ and using the finite propagation speed as in [B2, § lib)], we may 
assume that in Xf, we have the identification (TX)^y ~ R^^ with G R^^ representing 
xq and that the extended fibration over R^^ coincides with the given fibration restricted 
toE(0,e). 

Take any vector Y G R^ . We can trivialize K{T*B)®/S.®W by parallel transport 
along the curve u — ^ uY with respect to V'^. 



Let p{Y) be a C°^-function over R which is equal to 1 if |F| < |, and equal to if 
\Y\ > |. Let A"^'''- be the ordinary Laplacian operator on (TX)^^,. Let H^q be the vector 
space of smooth sections of the bundle {A(T* B)®A®W) x^ over (TX)^,). For t > 0, let 
L] be the operator acting on Hx^: 



;i.25) 



Ll = {1 - p\Y)){-tA^^) + p\Y)Al 



For t > 0, s G HxQ, we write 



;i.26) 



L? = Fi'LlF,. 



Let {ei,- ■ ■ ,621'} be an orthonormal basis of (TX^)^^, and let {e2i'+i,- ■ ■ ,e2k} be an 
orthonormal basis of Nx9/x,xo- Let Lf be the operator obtained from Lf by replacing 
the Clifford variables c(ej) with I < j < 21' by the operators -^ — \ft%ey 

-- 1,2,3 be the smooth kernel 



Let P/(F,F') with Y,Y' G (TX)^^ and \Y'\ < 



4' 



associated to exp(— LJ) with respect to the volume element dvrx^ 0^')- By using the 
finite propagation speed method, there exist c,C > 0, such that for Y G Nx3/x,xo, 

\Y\ < I and t g]0, 1], we have 



:i.27) 



Ptig-X Y)k{xo, Y) - Plig-'Y, Y) 



C 
< cexp(--). 



For a G C(e-', iej)(i<j<2r), let [a]™''^ G C be the coefficient of e^ A 
expansion of a. Then as in [B2, Proposition 11.12], if F G Nxa/x 



A e^'' in the 



1.28) Tr, 



gP,'{g-XY) 



-2i)2 



^ dim Xs ,- ^ dim Nxa / X ^y 



gPt 



g-^Y Y 



Let RTj^g, R^g,- ■ ■ be the corresponding restrictions of R^^,R^,--- to M^. Let 
Ve- be the ordinary differentiation operator on {TX)x„ in the direction Cj. By [ABoP, 
Proposition 3.7], and (1.20), we have, as t ^ 0, 



1 9 

;i.29) L? ^Ll==-J2 (Ve, + ^ {Rfj^.Y, e,> ) + R 



2k 

w 

\Ma- 



By proceeding as in [B2, §llg)- §lli)], we obtain the following: there exist some 
constants 7>0, c>0,C>0,rGN such that for t g]0, 1] and Y, Y' G {TX)^^, we have 



:l3o) 



Pi'{y, y) < c(i + |r I + |r 1)^ exp(-c|F - r n, 
{pf - p^){Y, Y') < cr(i + |r| + |r|)''exp(-c|r - rf ). 



From (1.28) and (1.30), we get 

lim*->o/ i.Ki <^Tv,[gPl{g-^Y,Y)]dv^^g^^{Y) 

YeNxg/x 

(1.31) =]im,^oLr^<.,sv-A-^^)'^''''^'''^^^s[gPh9-'Y,Y)]dv^x.fxiY) 

y<^Nx9/x 

= j^^^^^{-2iY^^'-^^'^Tv,[9Pl{9~'Y,Y)r^''dvMx.^xiy)- 
Now we define 

2fc 

(1-32) ^ = - E (^^. + 4 ^^JmoY, e^> 

By Mehler's formula [G], the smooth kernel q{Y,Y') for Y^Y' G TX, associated to 
exp(— y4) is given by 



:i.33) 



,(r,rO ^ (4.)-detV^(-£^) exp { - i (^^^^^^r,r 



' /^2\sinhri?^^/2) ' // 



4 \tanh(i?^^/2) ' / 2 \sinh(i?^^/2) 
From (1.9) and (1.33), we deduce, for Y G Nxs/x, 

i?^^/2 



q{g-XY) = {An)-''dee^^(-^ 



-^{-U;iil^(-Mi^^^^^ 



'134^'^" ' ^ ^ ^ Vsinh(i?^^/2) 

2 \ sinh(i?^/2) 
On the other hand, for Y G N{6), we have 



•^■»' (2i;n^^^«"^-^7-^;nii#Mi(^"''*-'-^'"''«'^-^->- 



It is easy to see that 

(1.36) cosh(i?^/2) - i(e«"/^^-^ + e-^''/'g) = ^(1 - 9-'){e^''^' - e^^'^'^g). 



From (1.9), (1.34)- (1.36), we get 






det^/^ 



sinh(i?^^72) 



|dimX9 



-ij^ 



We may and will assume that on the basis {em}2i'+i<m<2k-, the matrix of g has 
diagonal blocks 



cos 



— sm 



sml^jj cos^y^, 



, < ^j < vr. 

Then one verifies easily that the action of (? on A is given by 
(1.38) g = n,/+i<j<fc (^ cos{6j/2) + sin(6'j/2)c(e2j_i)c(e2j) 

By (1.29) and (1.38), we know that 



(1.39) Tr, [gPi^{g-X Y)\ = n,+i<,<fc ( - 2t sin(^,/2)j Tr [g exp{-Rjl,,)\ qig'X Y). 

From (1.24), (1.27), (1.31), (1.37) and (1.39), we finally arrive at the wanted formula 
(1.18). ■ 

By Theorems 1.2 and 1.3, we now have the complete proof of Theorem 1.1. ■ 



2 Family Rigidity Theorem 

This section is organized as follows. In Section 2.1, we state our main theorem of the 
paper: the family rigidity theorem. In Section 2.2, we prove it by using the equivariant 
family index theorem and the modular invariance. In Section 2.3, motivated by the 
family Witten rigidity theorem, we state a conjecture about a i^-theory level rigidity 
theorem for elliptic genera. 

Throughout this section, we use the notations of Section 1, and take G = S^. 

2.1 Family rigidity theorem 

Let TT : M ^ B he a fibration of compact manifolds with fiber X and dimX = 2k. We 
assume that the S^ acts fiberwise on M, and TX has an S'^-equivariant spin structure. 
As in [AH], by lifting to the double cover of S^, the second condition is always satisfied. 
Let V" be a real vector bundle on M with structure group Spin{2l). Similarly we can 
assume that V has an ^^-equivariant spin structure without loss of generality. 



The purpose of this part is to prove the elhptic operators introduced by Witten [W] 
are rigid in the family case, at least at the equivariant Chern character level. Let us 
recall them more precisely. 

For a vector bundle E on M, let 



(2-1) A /'Z7^ _ 1 I +77 I +2a2 



St{E) = l + tE + eS^E + ---, 
At{E) = l + tE + eA^E + ---, 



be the symmetric and exterior power operations in i^(M)[[t]]. Let 

e;(TX) = ®- iA,.(TX) ®-=i S,^iTX), 
(2.2) e,(TX) = ®- iA_^„-v2(TX) ®-=i S,^{TX), 

e_,(TX) = ®- iA^„-v2(TX) ®-=i V(^^)- 

We also define the following elements in if (M)[[g^/^]]: 

Q',iTX\V) = ®^=iA,.(\/) ^rn=i S.ATX), 
(^ ON Q,{TX\V) = ®- iA_,„-v2(V^) ®^=i V(^^), 

^ ^ 0-,{TX\V) = ®- iA^„-v2(V^) ®^=i V(^^), 

e*(TX|\/) = ®- iA_,.(\/) 0-^1 V(^^)- 

Let Pi(-)5i denote the first S'-'^-equivariant Pontrjagin class and A{V) = A'^{V) © A^(y) 
be the spinor bundle of V. 

In the following, we denote by D^ W the Dirac operator on A © W^ as defined 
in Section 1. We also write df = D^ Cg> A{TX). The following theorem is the family 
analogue of the Witten rigidity theorems as proved in [BT], [T] and [Liu2]. 

Theorem 2.1 (a) The family operators df®%{TX), D^®Qq{TX) and D^^Q_g{TX) 
are rigid. 

(b) IfpiiV)si =pi{TX)sr, thenD''®Aiy)®%{TX\V), D^® (A+(\/) - A-(\/))® 
0*(TX|\/), D^ ® Qq{TX\V) and D^ ® Q^q{TX\V) are rigid. 

1.1 Proof of the family rigidity theorem 

For r G H = {r G C; Imr > 0}, g = e^^^^, let 

e^{v, r) = c(g)n^^i(l + g"-i/2e2™)n^^i(l + qn^i/2^~2-^iv^^ 
(2 A] ^2{v,t} - c{q)ll^^^{l-q 'e jii„=i(l-g 'e j, 

0lv,T) = c(g)gi/S2sin(7rt;)n^^i(l - g"e2™)n^^i(l - g^e'^™). 

be the classical Jacobi theta functions [Ch], where c(g) = n^i(l — g"). 

Let g = e^'^** G S*^ be a generator of the action group. Let {M^} be the fixed 
sub manifolds of the circle action. Then tt : M^ -^ B he a submersion with fibre X^- We 
have the following equivariant decomposition of TX 

(2.5) TX|M„ = N,®---®Nh® TX„ 

10 



Here N^ is a complex vector bundle such that g acts on it by e^'^*™^*. We denote the 
Chern roots of A^-^ by {2TTixi^}, and the Chern roots of TXa (8>r C by {±2TTiyj}. We will 
write dime ^7 = d{m^) and dimX^ = 2ka. 
Similarly, let 



(2.6) 



M 



\Mo, 



VI© 



V, 



'o' 



be the equivariant decomposition of V restricted to M^. Assume that g acts on V^ by 
^2mn^t^ where some n^ may be zero. We denote the Chern roots of V^ by 2TTiu{j. Let us 
write dimR\/y = 2d{nv). 

For f{x) a holomorphic function, we denote by f{y){TX^) = Iljf{yj), the symmetric 
polynomial which gives characteristic class of TX^, and we use the same notation for 
A^^. Now we define some functions on C x H with values in H*{B), 



FdXt.r) 
Fnit^r) 



ZjqTT^ 



ZjrvTTj, 



2'Ky- 



yiy,^) 



)(TX^)n, 



0{y,r)J 

2ny^4^)iTXnIl,i^ 



^ 1 ^1(3:7 + m^t,r) -^ 



F^D(t,T) = T.a'n-^ 



27iy 



Oiy,r). 

03{y,r 



{TX^)n 



(2.7) 



Fl{t,r)- 
FUt,T)- 

FUt.r)- 



V '1 * 






(x^ + m^t, r) 

6{x^ + m^t, r) 

^\ 6{xy + m^tjT) 
Il^9i{u^ + n„t,r)(V;) 



(AT, 






(A^, 



(a;^ + m^t,T){Nj) 



;-k 



ZjqTT^ 



2niy 



{TX^ 



) (TX« 



n^^(a;^ + m-^t, r) (A^-y) . 
n^^3(M^ + n„t,r)(V;) 



;-k+l 



zjQ,7r^ 



Lv%,r)y 



^(l/,r) 



11^6' (X-y + m^t, T){Ny 



By Theorem 1.1 and [LaM, p238], we get, for t G [0, 1] \ Q and g = e 



2iTit 



f2.8) 



FdXt.r) = c\{lnd{df®%{TX))y 

FdH^t) = q-''/''chg(lnd{D^ ® Qq{TX)) 

F_o{t,T) = q-''/\hg(lnd{D^ ® 0_,(TX))], 

Fj^(t,r) = c(g)^-V~''^/^chJlnd(D^ ® A(\/) ® Q'g{TX\V))\ 



^ D 



it.r) 



FYn{t,T 



c(^qy-kq-k/8^Yig(lnd{D^ (g) Qq{TX\V)) 
c(g)'^V''/^chJlnd(D^ (g) Q^g{TX\V)) 
F^,{t,T) = (-l)'c(g)^- V^/'chg(lndp^ ® (A+(V^) - A"(V^)) 

®0*(TX|\/))). 



Considered as functions of (t,r), we can obviously extend these F's and F^'s to 
meromorphic functions on C x H. Note that these functions are holomorphic in r. The 
rigidity theorems are equivalent to the statement that these F's and F^'s are independent 
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of t. As explained in [Liu2], we will prove it in two steps: i) we show that these F, F^ 
are doubly periodic in t; ii) we prove they are holomorphic in t. Then it is trivial to see 
that they are constant in t. 

Lemma 2.1 (a) For a,b E 2Z, Fd^(t, r), Fcit, r) and F^^it, r) are invariant under the 
action 

(2.9) U -.t-^t + ar + b 

(h) Ifpi{V)si = Pi{TX)si, then Fl{t,T),F^{t,r), F'[^{t,r) and F^.{t,T) are m- 
variant under U . 

Proof: Recall that we have the following transformation formulas of theta-functions 

[Ch]: 

e{t + i,T) = -e{t, t), e{t + t,t) = -q-'/^e-^^'%t, t), 

^i(t+l,r) = -^i(t,r), ^i(t + r,r) = g-i/2e-2-*^i(t,r), 

^^•'^^ ^2(t + 1, r) = e^it, r), e^it + t,t) = -q-^/'e-'-'^%it, r), 

^3(t + 1, r) = ^3(t, r), 9sit + t,t) = q-'/^e-^-^%{t, r). 

From these, for 6^ = 6, 6i, 62, 63 and (a, b) G (2Z)^, / G Z, we get 

(2.11) e^{x + l{t + aT + b),T)= e-'^i(2'«^+2''"*+''"'^)^^(x + It, r) 

which proves (a). 

To prove (b), note that, since ^1(^)51 = pi(TX)s^, we have 

(2.12) ^v,M + n,tf = S,-(|/,)' + S^,i(4 + m^tf. 
This implies the equalities: 

/„ -| qN ^v,j'^v"'v 2^-yjTll^X^, 

which together with (2.11) proves (b). ■ 

For g = \ ^ ) ^ SL2{7i), we define its modular transformation on C x H by 

, t ar + b 
(2.14) g{t,r 



CT + d CT + d 
The two generators of 5*^2 (Z) are 

(2.15) S=(J -o)-^={l \ 

which act on C x H in the following way: 

t 1 



(2.16) S{t, r) = (-, --) , T(t, r) = (t, r + 1). 

Let \E'r be the scaling homomorphism from A(T*B) into itself : /3 -^ r^'^'^^^p. 



12 



Lemma 2.2 (a) We have the following identities: 

...7^ F,^{^^,-'-)=t'^rFDit,r), F,Xt,r + l) = F,^it,T), 

^ ' F_z.(^-i)=^'=^.F-D(t,r), Fnit,r + l) = F_nit,T)e'^^\ 

(b) Ifpi{V)s^ =j9i(TX)si, then we have 



(2.18) F_ 



Fli^,-^) 
FY It 1, 



D*Vt-? 



'^^'^rFy{t, t), Flit, r + 1) = e"f ('^-^^^(t, r), 
)'-^t'^^rF':^{t, r), F^{t, r + 1) = e-^^'FYo{t, r), 



2 2 



r . 



Proof: By [Ch], we have the following transformation formulas for the Jacobi theta- 
functions: 



■ t 1 



(2.19) 






« V J 



^e T 



-e T 



-e r 



-e{t,T), 9{t,T + l)=e^9{t,r), 

-^2(t,r), ^i(t,r + l) = ex^i(t,r), 

'Mt,T), e2it,T + l) = 9s{t,T), 

-^3(t,r), ^3(t,r + l) = ^2(t,r). 



The action of T on the functions F and F^ are quite simple, and we leave the proof 
to the reader. Here we only check the action of S. By (2.19), we get 



t 1 



(2.20) 



r r 



zjQ,7r^ 









9{Ty,T) 



(rx-y + m^t, r) 



If a is a differential form on B, we denote by {a}*^^-' the component of degree p of a. It 
is easy to see that (2. 17) for F^^ follows from the following identity: 



r ''"' < TT* 



f2.211 



^ /£^'!? ) jTxnu, (r ^ ^;/""^ + "^^^' ;^ ) (iv,)^ ' ''"' 



Oiry, 
Oiiy,T) 



{jXy + m^t, r) 
_;^^i(a;^ + m-yt, r) 



y— -— )(Txg)nJ2-^ ";;'^-^ ' ••"•^^'•M (iv^, 

^e{y,T)J^ ' ^\ e{xy + myt,T^ '^ ^' 



(2p) 



By looking at the degree 2(p + ka) part, that is the (p + A;a)-th homogeneous terms of 
the polynomials in x's and |/'s, on both sides, we immediately get (2.21). 
From (2.7), (2.20) and (2.21), we obtain 



(2.22) 



{F,,(-,-i)}(2rt=,V{Fz,(t,r)}(2p), 
r r 



which completes the proof of (2.17) for Fd^. The other identities in (2.17) can be verified 
in the same way. 

By using (2.12), (2.19) and the same trick as in the proof of (2.17), we can obtain 
the identities in (2.18). This completes the proof of Lemma 2.2. ■ 

The following lemma implies that the index theory comes in to cancel part of the 
poles of the functions F's and F^'s. 
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Lemma 2.3 IfTX and V are spin, then all of the F's and F^ 's above are holomorphic 
in {t, t) for {t, r) G R x H. 

Proof: Let z = e^'^** and /' = dimM. We will consider these F's and F^'s as 
meromorphic functions of two complex variables {z, q) with values in H*{B). 
i) Let N = niaxQ,^^|?72^|. Denote by D^r C C^ the domain 

(2.23) \q\^/^ < \z\ < \q\-^/^,0< \q\ < 1. 

Let fa be the contribution of the component M" in the functions F's and c(g)^~'F^'s. 
Then in Dj^, by (2.4), (2.7), it is easy to see that fa has expansions of the form 

(2.24) q-''/^U^{z"''' - 1 )-''"'('"-' )S^^i6c,,„ (-2) g", 

where a is an integer and ha{z,q) = ^'^=iba,n{z)q'^ is a holomorphic function of {z,q) G 
D^, and ba,n{z) are polynomial functions of z. So as meromorphic functions, these F's 
and c(g)'^~'F^'s have expansions of the form 

(2.25) g-"/sS^^i6„(2)g". 

with bn{z) rational function of z, which can only have poles on the unit circle |z| = 1 C 

Now, if we multiply these F's and c(g)'^^'F^'s by 

(2.26) f{z) = n«n^(l - 2'»7)«'d("^7)^ 

we get holomorphic functions which have convergent power series expansions of the form 

(2.27) g-«/8S-iC„(2)g". 

with {cn{z)} polynomial functions of z in D^. 

By comparing the above two expansions, we get for n G N, 

(2.28) Cn{z) = f{z)bn{z). 

ii) On the other hand, we can expand the Witten element O's into formal power 
series of the form S^^oi?„g" with K e K{M). So, for t G [0, 1] \ Q, z = e^'^**, we get a 
formal power serie of q for these F's and c(g)'^~'F^'s: 

(2.29) g'"/SS^^och^(Ind(F'^ ® i?n))g" 

with a G Z. 

By (1.6), we know 

(2.30) ch,(ind(D^ (g) i?„)) = s;;;2^(„)a„,„z-. 

with am,n £ H*{B), and N{n) some positive integer depending on n. 
By comparing (2.7), (2.25) and (2.30), we get for t G [0, 1] \ Q, z = e 

(2.31) On[Zj = L^^_^^^^am,n^ • 
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2nit 



Since both sides are analytic functions of z^ this equahty holds for any 2; G C. 
By (2.28), (2.31), and the Weierstrass preparation theorem, we deduce that 

(2.32) g-"/^S- ,6„(^)g" = J-^g-Z^s- ,c„(z)g^ 

is holomorphic in D^. Obviously R x H lies inside this domain. The proof of Lemma 
2.3 is complete. ■ 

Proof of the family rigidity theorem for spin manifolds: We will prove that these F's 
and F^'s are holomorphics on C x H, which implies the rigidity theorem we want to 
prove. 

We denote by F one of the functions: F's, F^'s, \E'^F's and \1',-F^'s. From their 
expressions, we know the possible polar divisors of F in CxH are of the form t = 'j{cT+d) 
with n, c, d, I intergers and (c, rf) = 1 or c = 1 and d = 0. 

We can always find intergers a,b such that ad ~ he = 1. Then the matrix g = 

G SL2(Z) induces an action 



— c a 

t dr-b 



(2.33) Figit,T)) = F(^- 



-CT + a —CT + a 



Now, if t = j(cr + (i) is a polar divisor of F(t, r), then one polar divisor of F{g(t, r)) is 
given by 

/r^ o.\ t n / dr — b , 

(2.34) = T c + d 

~CT + a l \ —CT + a 

which exactly gives t = n/l. 

But by Lemma 2.2, we know that, up to some constant, F{g{t, r)) is still one of these 
F's, F^'s, \E'^F's and \1't-F^'s. This contradicts Lemma 2.3, therefore s completes the 
proof of Theorem 2.1. ■ 

2.3 A conjecture 

Motivated by the family rigidity theorem. Theorem 2.1, we and Zhang would like to 
make the following conjecture 

Conjecture: The operators considered in Theorem 2.1 are rigid on the equivariant 
K -theory level. 

This means that, as elements in Kg{B), the equivariant index bundles of those elliptic 
operators actually lie in K{B). Note that this conjecture is more refined than Theorem 
2.1, since the equivariant Chern character map is not an isomorphism. In [Z], Zhang 
proved this for the canonical S'pm'^-Dirac operator on almost complex manifolds. 
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O Jacobi forms and vanishing theorems 

In this Section, we generalize the rigidity theorems in the previous Section to the nonzero 
anomaly case, from which we derive a family of holomorphic Jacobi forms. As corollaries, 
we get many family vanishing theorems, especially a family il- vanishing theorem for loop 
space. This section generalizes some results of [Liu4, §3] to the family case. 

This section is organized as follows: In Section 3.1, we state the generalization of the 
rigidity theorems to the nonzero anomaly case. In Section 3.2, we prove this result. In 
Section 3.3, as corollaries, we derive several family vanishing theorems. 

We will keep the notations of Section 2. 

3.1 Nonzero anomaly 

Recall that the equivariant cohomology group Hgi{M, Z) of M is defined by 

(3.1) H*^{M,Z) = H*{MxsiES\Z). 

where ES^ is the universal S^-principal bundle over the classifying space BS^ of S^. So 
H*gi (M, Z) is a module over H* {BS^ , Z) induced by the projection W: Mxgi ES^ -^ BS\ 
Let pi{V) si , Pi{TX) s^ G Hgi{M, Z) be the equivariant first Pontrjagin classes of V and 
TX respectively. Also recall that 

(3.2) H*{BS\Z) = Z[[u]] 

with u a generator of degree 2. 

In this section, we suppose that there exists some integer n G Z such that 

(3.3) p,iV)si - PiiTX)si = n ■ W*u\ 

As in [Liu4], we call n the anomaly to rigidity. 

Following [Liu2], we introduce the following elements in i^(M)[[g^/^]]: 



(3.4) 



0;(TX|\/), = ®^=iAgn(\/ - dim V) ®^^i Sgr^iTX - dimX), 
Qg{TX\V), = ®^^,A_^„_i/2(l^ - dimV^) 0^^^ Sg^{TX - dimX), 
0_,(TX|\/), = ®^^iA^„_i/2(l^ - dimV^) ®^^i Sg^iTX - dimX), 
Ql{TX\V), = ®^=iA_,n(\/ - dimV) ®^^i Sg^{TX - dimX). 

For g = e^'^**, q = e^'^*'^, with (t, r) G R x H, we denote the equivariant Chern charac- 
ter of the index bundle of D^®A(\/)®e;(TX|\/)^, D^®eg(TX|\/)^, D^®e_g(TX|\/)^, 
and D^ ® (A+(\/) - A-{V)) ® e;(TX|\/), by 2'FX,,(t, r), F^^„(t, r), F_^^,,(t,r) and 
(— 1)^F^* ^{t, t) respectively. Similarly, we denote by H{t, r) the equivariant Chern char- 
acter of the index bundle of 

D ® ^^^^Sg^{TX - dimX). 

Later we will consider these functions as the extensions of these functions from the unit 
circle with variable e^'^** to the complex plane with values in H*{B). For a a differential 
form on B, we denote by {a}^^^ the degree p component of a. 

The purpose of this section is to prove the following theorem which generalizes the 
family rigidity theorems to the nonzero anomaly case. 
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Theorem 3.1 Assume ^1(^)51 — pi{TX)si = n ■ 'k*u'^ with n E 7^. Then for p G N, 
{Fj^^^(t,r)}(2p), {F^,^(t,r)}(2p), {^^^^^(t, r)}(2p) are holomorpic Jacobi forms of index 
n/2 and weight k + p over (2Z)^ x F with T equal to ro(2), r°(2), Fg respectively, and 
{^D* t)(^' t)}*-^^^ is a holomorphic Jacobi form of index n/2 and weight k — I + p over 
(2Z)2 X SL2{Z). 

See Section 3.2 for the definitions of these modular subgroups, Fo(2), F°(2) and Fg. 

3.2 Proof of Theorem 3.1 

RecaU that a (meromorphic) Jacobi form of index m and weight / over L xi F, where 
L is an integral lattice in the complex plane C preserved by the modular subgroup 
F C 5*^2 (Z), is a (meromorphic) function F(t, r) on C x H such that 



(3.5) CT + d' CT + d 
Fit + Ar + /i, r) = e-2-™(^'-+2At)^(^^ ^^^ 

where (A, /i) G L, and g = \ , J G F. If F is holomorphic on C x H, we say that F 

is a holomorphic Jacobi form. 

Now, we start to prove Theorem 3.1. Let g = e^^^** G S*"*^ be a generator of the action 
group. For a = 1,2, 3, let 

(3.6) 9'i0,r) = ^^(t,r)|,=o, ^^(0, r) = ^,(t, r)|i=o 



By applying Theorem 1.1, we get 

(0,r) 
/i(0,r 



FlA^^r) = i2nr'^^Flit,r) 



FlJt,r) = {2n)-'^-^^FUt,r), 



^2(0, r) ^ 
^'•^^ F~D,At. r) = {2.)-'^^^FYo{t, r), 

F^.,,(t,r) = (27r)'V(0,V)'=-'F^.(t,r), 

2my v^^^, e'(Q,T) 



H(t,r) = (27r^)-'=S«7r.[(-^)(TX 



\9{y,T)J Il^9{x^ + m^t,T){N^) 

Lemma 3.1 Ifpi{V)s^ — pi{TX)s^ = n-W*u'^, we have 

FlA^, -i) = T^e--^''l^^>rFl^{t, t), Fl^it, r + 1) = Fl^{t, r), 

(3.8) F_V(7> -'-) = r^e-'-^'I^^^FY^^^it, r), F^^it, r + 1) = Fl'^,,(t, r), 

Ifpi{TX)si = -n-7r*u^, then 

(3.9) /7(-,-i) = r'=e™*'/"^,/J(t,r), iJ(t, r + 1) = i7(t, r). 

r r 
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Proof: First recall that the condition on the first equivariant Pontrjagin classes 
implies the equality 



n-r 



(3.10) KjK + n,ty - [^.{yjY + ^j,jixl^ + m^ty 

which gives the equalities: 



(3.11) 









■i^2 



The action of T on the functions F and F^ are quite easy to check and we leave the 
detail to the reader. We only check the action of S. By (2.7), (2.19), (3.7) and (3.11), 
we have 






t' 



(3.12) 



^(x'^^ 



(27ri)-^ 
^'(0,-i)^' 



L^i(0,-i)'V%,-iV 

k 



2Txiy 



^){TX^ 



(2vr.)-V^e-(-V.)S„., [ W^) / 2vrz, x ^^^,^ H 



'n^^(a;^ + m^^,-i)(Ar, 



7/ 



^i(0,r)'V^(ry,r) 



' n^6'(rx^ + m^t, r) (A^^) 



As in (2.21), by comparing the (p + A;Q,)-th homogeneous terms of the polynomials in x's, 
y's and m's on both side, we find the following equation 



(3.13) 



TT* ( 



/ 2Txiy 



d{ry,T)J 



){TX^ 



r^TT, 



I1^9{tXj + rrijt, t){N^)- 



(2p) 



^(2p) 



By (3.12), (3.13), we get the equation (3.8) for Fj^^. We leave the other cases to the 
reader. ■ 



Recall the three modular subgroups: 
a b 




G5L2(Z)|c = 0(mod2)^, 
eSL2iZ)\b = 0{mod2)\, 
e SL^iZ) 



a b 
c d 



1 
1 



or 



1 

1 



[mod2) 



Lemma 3.2 If pi{V)si - pi{TX)si = n ■ vfV, then for p E N, {Fj^,„(t, r)}(2p) is 
a Jacohi form over {Tlf -a ro(2); {F^,^(t, r)}(2p) is a Jacobi form over {TLf yx r°(2); 
{F_^^^„(t,r)}(2p) isaJacobiformover(:XLfy\Ye. Ifpi{TX)si = -nyfV, then {H{t,T)Y^P^ 
is a Jacobi form over (2Z)^ xi 5*1/2 (Z). All of them are of index ^ and weight k + p. 

The function {F)^, ,^{t, r)}*^^^-' is a Jacobi form of index ^ and weight k — I + p over 
(2Z)2 X SL2{Z). 
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Proof: By (2.19), (3.7), we know that these F^'s and H satisfy the second equation 
of the definition of Jacobi forms (3.5). 

Recall that T and ST'^ST generate ro(2), and also r°(2) and Tq are conjugate to 
ro(2) by S and TS respectively. By Lemma 3.1, and the above discussion, for F^'s and 
H, we easily get the first equation of (3.5). ■ 

For g = \ ^ I £ SL2{7j), let us use the notation 



c d 

I -2mract^/(cT+d)-p( ^ ^'^ 



(3.15) F{g(t,T))\^^i = {cT + rf)-'e-""*™"* /l^r+dj^^ 



CT + d CT + d' 



to denote the action ol g on a, Jacobi form F of index m and weight /. 

By Lemma 3.1, for any function in F G {{F^}*^^^\ i^*^^^)}, its modular transforma- 
tion {F}(2p)((^(t,r))|n^fc+p (or {F}(2p)((^(t, r))|n^,_,+p) is still one of the {{F^}(2p)}'s and 
H^'^P\ Similar to Lemma 2.3, we have 

Lemma 3.3 For any g G SL2{Z), let F{t,T) be one of the {F^j^^rt 's or H^^p\ Then 
F{g(t, r))|ii,fc+p is holomorphic in (t, r) for t G R and r G H. 

As in Lemma 2.3, this is the place where the index theory comes in to cancel part of 
the poles of these functions. Of course, to use the index theory, we must use the spin 
conditions on TX and V. 

Now, we recall the following result [Liu4, Lemma 3.4]: 

Lemma 3.4 For a (meromorphic) Jacobi form F{t,T) of index m and weight k over 
L X r, assume that F may only have polar divisors of the form t = {cr + d)/l m C x H 
for some integers c,d and I ^ 0. If F{g(t,T))\m,k is holomorphic for t G R, r G H for 
every g G 5*^2 (Z), then F(t, r) is holomorphic for any t G C and r G H. 

Proof of Theorem 3.1: By Lemmas 3.1, 3.2, 3.3, we know that the {F^}*^^^^'s and 
i7*^^^) satisfy the assumptions of Lemma 3.4. In fact, all of their possible polar divisors 
are of the form / = {ct + d)/m where c, d are integers and m is one of the exponents 
{rrij}. The proof of Theorem 3.1 is complete. ■ 

3.3 Family vanishing theorems for loop space 

The following lemma is established in [EZ, Theorem 1.2]: 

Lemma 3.5 Let F be a holomorphic Jacobi form of index m and weight k. Then for 
fixed T, F{t, t), if not identically zero, has exactly 2m zeros in any fundamental domain 
for the action of the lattice on C. 

This tells us that there are no holomorphic Jacobi forms of negative index. Therefore, if 
m < 0, F must be identically zero. If m = 0, it is easy to see that F must be independent 
oft. 

Combining Lemma 3.5 with Theorem 3.1, we have the following result. 
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Corollary 3.1 Let M, B, V and n be as in Theorem 3.1, Ifn = 0, the equivariant Chern 
characters of the index bundle of the elliptic operators in Theorem 3. 1 are independent 
of g E S^ . If n < 0, then these equivariant Chern characters are identically zero; in 
particular, the Chern character of the index bundle of these elliptic operators are zero. 

Another quite interesting consequence of the above discussions is the following family 
il-vanishing theorem for loop space. 

Theorem 3.2 Assume M is connected and the S^-action is nontrivial. If pi{TX)s^ = 
n ■ W*u'^ for some integer n, then the equivariant Chern character of the index bundle, 
especially the Chern character of the index bundle, of D ® ®^^-^^Sqm,{TX — dimX) is 
identically zero. 

Proof: In fact, by (3.11), we know that 

(3.16) Tjjm^-d{mj) = n. 

So the case n < can never happen. If n = 0, then all the exponents {rrij} are zero, 
so the S'^-action cannot have a fixed point. By (2.7)and (3.7), we know that H[t,T) is 
zero. For n > 0, one can apply Lemmas 3.1, 3.4 and 3.5 to get the result. ■ 

As remarked in [Liu4], the fact that the index of D ^ ^'^^iSqm[TX — dimX) is zero 
may be viewed as a loop space analogue of the famous il-vanishing theorem of Atiyah 
and Hirzebruch [AH] for compact connected spin manifolds with non-trivial S'^-action. 
The reason is that, this operator corresponds to the Dirac operator on loop space LX, 
while the condition on pxiTX^gi is a condition for the existence of an equivariant spin 
structure on LX. This property is one of the most interesting and surprising properties 
of loop space. Now, under the condition of Theorem 3.2, a very interesting question is 
to know when the index bundle of this elliptic operator is zero in K{B). 
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